1. Field of the Invention
The present invention relates to verification of integrated circuits, and more particularly to performing efficient continuous grading flow for test coverage analysis.
2. Description of the Related Art
Verification of complex integrated circuit chips such as microprocessors often requires large sets of tests to be executed to verify the quality of a design-under-test (DUT). To use compute resources efficiently, it is important to select the set of tests that most efficiently verifies the entire design. Functional coverage metrics are used to measure how well a test verifies the DUT. To determine the most efficient set of tests, test grading, a process that compares a test coverage footprint of each test, is performed.
Running and grading a large set of tests can be expensive. Many hours of computation are often required for both the simulation and grading steps. The grading process itself can require a large amount of disk space and memory. The resources required to grade a set of tests increase exponentially with respect to the number of tests being graded.
For example with certain complex integrated circuit designs, a weekly regression run can include tens of thousands of tests. These tests vary in length and complexity, taking anywhere from a few hours to a few days to run. Additional tests are added to the run during project development as features are added and new requirements are defined. Running the entire set of tests to qualify every change can be expensive and time consuming.
To reduce the time to qualify localized changes, it is known to run a subset of the entire regression run. Test grading is the process of identifying a subset of tests that best covers the DUT as measured by the coverage metrics. Known methods for grading tests can require waiting for all tests in a regression run to completion, followed by a relatively slow and memory intensive process of grading the entire set of tests at once. Results are not available until the entire process has completed, and all of the test data has to be retained during the entire procedure. With known methods for regression testing, the entire set of regression tests is executed to completion, collecting the coverage data from all tests. Additionally, with a procedure that is sometimes referred to as greedy grading, the coverage result for each test is compared with the results from every other test in the set to determine what parts of the design are uniquely covered by a particular test.
Another known approach is to perform another grading technique referred to as quick grading, which works by grading each incoming test relative to the aggregate coverage of all the previously graded tests in some pre-determined order. This approach is significantly faster than greedy grading, but can only provide equally accurate results if the tests that provide the most unique coverage are graded first. However, it is extremely difficult to determine a proper order without running greedy grading.